Information processing apparatus, information processing method, and program

ABSTRACT

An information processing apparatus including an imaged image input unit inputting an imaged image of a facility imaged in an imaging device to a display control unit, a measurement information input unit inputting measurement information measured by a sensor provided in the facility from the sensor to a creation unit, a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input by the measurement information input unit, and a display control unit overlaying and displaying the virtual image created in the creation unit and the imaged image input by the imaged image input unit on a display device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of International Application No.PCT/JP2011/057769, with an international filing date of Mar. 29, 2011,which designates the United States of America, which claims priority toJapanese Application No. 2010-079219, filed on Mar. 30, 2010, JapaneseApplication No. 2010-138517, filed on Jun. 17, 2010, JapaneseApplication. No. 2011-061339, filed on Mar. 18, 2011, and JapaneseApplication No. 2011-066594, filed on Mar. 24, 2011, the entire contentsof each of which are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to an information processing apparatus, aninformation processing method, and a program.

BACKGROUND ART

Conventionally, in an observation trip regarding an installation for afactory, or the like, observers observe an actual installation orfacility, or observe a prepared panel or video. On the other hand, forexample, Patent Document 1 proposes a system to overlay-display elementinformation corresponding to the site on an actual site image imaged bya portable terminal or the like, according to the movement of anobserver.

In dangerous worksite areas, e.g., at a factory or a construction site,the dangerous area is indicated, with a visual warning such as paint, asign or other posted warning. Since indications of these dangerous spots(dangerous spot indications) are stationery, in general, indications andsigns are posted throughout the range where there may be a danger.

There may be cases where a dangerous area is always dangerous, and theremay be cases where it is dangerous only during a particular period oftime. For example, crane operation site in a factory, the area below ahanging load becomes a dangerous spot only when the crane is inoperation.

CITATION LIST Patent Literature

-   Patent Literature 1: Japanese Laid-open Patent. Publication No.    2004-102835-   Patent Literature 2: Japanese Laid-open Patent Publication No.    2002-7703

SUMMARY Technical Problem

Depending on the installation or factory, there may be cases where thereis a danger and an observer cannot get close to a facility. Further,even when it is possible to get close, there may be cases where theinside of the facility cannot be seen.

For example, when the technology of Patent Document 1 is applied to anobservation trip in an installation or a factory, it is merely to theextent to see the name of a facility or the like as a virtual image, andthere has been a problem that the observer does not feel the reality ofthe observed installation or factory.

Further, in a factory, there may be cases where the operational statusof a facility is predicted by simulation, but the simulation results arejust displayed in a two-dimensional cross-sectional view on a screen ofa computer terminal. Accordingly, there has been a problem that it isnot easy for the operator to comprehend the simulation results.

Aspects presented herein are made in view of such problems, and a firstobject thereof is to communicate the operational status of a facility,or the like, or simulation results with real to an observer or worker inan installation or a factory.

Regarding a conventional dangerous spot indication for a stationerycrane arm or the like, for example, the entire movable range of thecrane arm is assumed to be a dangerous spot, and as dangerous spotindication (such as hatching or the like) is provided on the ground.Thus, in the conventional dangerous spot indication, there is a problemthat a dangerous spot is indicated as being larger than necessary,making it difficult to determine the actual range of the dangerous spot.Further, the conventional dangerous spot indication also has a problemthat dust, oil, or the like may soil the indication, making theindication itself unseeable, or maintenance such as cleaning orrepainting is required when it is soiled.

On the other hand, regarding a conventional dangerous spot indicationfor a mobile crane truck or the like, since the location where the craneis used is not stationery, an indication “KEEP CLEAR OF 10 M RADIUS” orthe like is posted on the crane truck. However, there is a problem thatthe dangerous spot cannot be visually recognized, and the dangerous spotbecomes obscure. Further, there are also cases to physically hinderentrance with pylons or the like, but there a problem that it takeslabor to prepare them.

Aspects presented herein are made in view of such problems, and a secondobject thereof is to indicate a dangerous spot more appropriately whenthe dangerous spot changes temporally.

Solution to Problem

Accordingly, an information processing apparatus, according to aspectspresented herein, has: an imaged image input unit inputting an imagedimage of a facility imaged in an imaging device to a display controlunit; a measurement information input, unit inputting measurementinformation measured by a sensor provided in the facility from thesensor to a creation unit; a creation unit creating a virtual imagerepresenting a status of an outside or inside of the facility based onthe measurement information input by the measurement information inputunit; and a display control unit overlaying and displaying the virtualimage created in the creation unit and the imaged image input, by theimaged image input unit on a display device.

With such a structure, the operating status of a facility, or the like,can be communicated in a realistic manner to an observer in aninstallation or a factory.

Note that the information processing apparatus corresponds to, forexample, an AR server which will be described later.

Further, an information processing apparatus, according to aspectspresented herein, has: a measurement in input unit inputting measurementinformation measured by a sensor provided in the facility from thesensor to a creation unit; a creation unit creating a virtual imagerepresenting a status of an outside or inside of the facility based onthe measurement information input by the measurement information inputunit; and a display control unit overlaying and displaying the virtualimage created in the creation unit on the facility which is seen througha display device.

With such a structure, the operating status of a facility or the likecan be communicated in a realistic manner to an observer in aninstallation or a factory.

Note that the information processing apparatus corresponds to, forexample, an AR server which will be described later.

Further, an information providing apparatus, according to aspectspresented herein, comprises an information providing apparatus having adisplay unit and connected communicably to a storage unit, theinformation providing apparatus having: a reading unit reading dangerousrange information indicating a dangerous range of each state of adangerous target and dangerous target position information indicating aposition of the dangerous target, which are stored in the storage unit;a position information obtaining unit obtaining apparatus positioninformation calculated from information indicating a position of theinformation providing apparatus which is detected in a positiondetection device; a direction information obtaining unit obtainingdirection information calculated from information indicating a directionof the information providing apparatus which is detected in a directiondetection device; a posture information obtaining unit obtaining postureinformation calculated from information indicating a posture of theinformation providing apparatus which is detected in a posture detectiondevice; a determination unit determining a visual field of theinformation providing apparatus based on the apparatus positioninformation obtained in the position information obtaining unit, thedirection information obtained in the direction information obtainingunit, the posture information obtained in the posture informationobtaining unit, and visual field information of the informationproviding apparatus which is defined in advance; and a display controlunit determining a dangerous target included in the visual field, basedon the apparatus position information, the direction information, theposture information, and the dangerous range information and thedangerous target position information which are read in the readingunit, and generating a dangerous range image for the determineddangerous target and displaying the image on the display unit.

Here, the “reading unit” corresponds to, for example, a retrieval unit1055 which will be described later. The “position information obtainingunit” corresponds to, for example, a measurement unit 1040 which will bedescribed later. The “direction information obtaining unit” correspondsto, for example, a direction information obtaining unit 1045 which willbe described later. The “posture information obtaining unit” correspondsto, for example, a posture information obtaining unit 1050 which will bedescribed later. The “determination unit” corresponds to, for example, avisual field determination unit 1060 which will be described later. The“display control unit” corresponds to, for example, a display controlunit 1065 which will be described later.

Advantageous Effects

According to aspects presented herein, the operating status of afacility or the like can be communicated in a realistic manner to anobserver in an installation or a factory.

Further, according to aspects presented herein, a dangerous spot can beindicated more appropriately when the dangerous spot changes temporally.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a system structure of afacility guidance system of Embodiment 1.

FIG. 2 is a diagram illustrating an example of a hardware structure ofan AR server.

FIG. 3 is a diagram illustrating an example of a software structure ofthe AR server of Embodiment 1.

FIG. 4 is a diagram (1) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 1.

FIG. 5 is a diagram (2) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 1.

FIG. 6 is a diagram (3) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 1.

FIG. 7 is a flowchart illustrating an example of display controlprocessing in the AR server of Embodiment 1.

FIG. 8 is a diagram illustrating a system structure of a facilityguidance system of Embodiment 2.

FIG. 9 is a diagram illustrating an example of a software structure ofthe AR server of Embodiment 2.

FIG. 10 is a diagram (1) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 2.

FIG. 11 is a flowchart illustrating an example of display controlprocessing in the AR server of Embodiment 2.

FIG. 12 is a flowchart illustrating an example of display controlprocessing in an AR server of Embodiment 3.

FIG. 13A is a diagram (1) illustrating an example of displaying afacility by each display layer.

FIG. 13B is a diagram (2) illustrating an example of displaying afacility by each display layer.

FIG. 13C is a diagram (3) illustrating an example of displaying afacility by each display layer.

FIG. 13D is a diagram (4) illustrating an example of displaying afacility by each display layer.

FIG. 14 is a diagram illustrating an example of a system structure of afacility guidance system of Embodiment 5.

FIG. 15 is a diagram illustrating an example of a software structure ofan AR server of Embodiment 5.

FIG. 16 is a flowchart illustrating an example of display controlprocessing in the AR server of Embodiment 5.

FIG. 17 is a diagram illustrating an example of a system structure of anoperation support system of Embodiment 6.

FIG. 18 is a diagram illustrating an example of a software structure ofthe operation support system of Embodiment 6.

FIG. 19 is a flowchart illustrating an example of processing in theoperation support system of Embodiment 6.

FIG. 20 is a diagram illustrating an example of a system structure of anoperation support system of Embodiment 7.

FIG. 21 is a diagram illustrating an example of a facility markercorrespondence table.

FIG. 22 is a diagram illustrating an example of a facility usage statustable.

FIG. 23 is a diagram illustrating an example of a software structure ofan HMD with camera.

FIG. 24 is a flowchart illustrating an example of processing in theoperation support system of Embodiment 7.

FIG. 28 is a diagram illustrating an example of a system structure ofthe operation support system of Embodiment 7.

FIG. 26 is a diagram illustrating an example of a structure of aninformation providing system.

FIG. 27 is a diagram illustrating an example of a hardware structure ofan AR providing apparatus.

FIG. 28 is a diagram illustrating an example of a functional structureof the AR providing apparatus.

FIG. 29 is a diagram illustrating an example of a hardware structure ofan information processing apparatus.

FIG. 30 is a diagram illustrating an example of a table storinginformation related to dangerous objects.

FIG. 31 is a diagram illustrating an example of a table storinginformation related to dangerous objects.

FIG. 32 is a diagram illustrating an example of a flowchart related tooperating state setting processing.

FIG. 33 is a diagram illustrating an example of a flowchart related, tooperating state setting processing.

FIG. 34 is a diagram illustrating an example of a flowchart related toposition information setting processing.

FIG. 35 is a diagram illustrating an example of a flowchart related todisplay processing.

FIG. 36 is a diagram illustrating an example of an AR visual field.

FIG. 37A is a diagram illustrating an example when a three-dimensionalaugmented image is displayed.

FIG. 37B is a diagram illustrating an example when a two-dimensionalaugmented image is displayed.

FIG. 37C is a diagram illustrating an example when an augmented image isdisplayed.

FIG. 37D is a diagram illustrating an example when an augmented image isdisplayed.

FIG. 38 is a diagram illustrating an example of a flowchart related todisplay processing.

FIG. 39 is a diagram illustrating an example of display when a dangerousobject is approaching.

FIG. 40 is a diagram illustrating a flowchart related to displayprocessing.

FIG. 41 is a diagram illustrating an example of a table storinginformation related to dangerous objects.

FIG. 42 is a diagram illustrating an example of a flowchart related todisplay processing.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments will be described based on the drawings.

Embodiment 1

FIG. 1 is a diagram illustrating an example of a system structure of afacility guidance system of Embodiment 1. As illustrated in FIG. 1, inthe facility guidance system, an AR (Augmented Reality) server 1, acamera 2, sensors 3 ₁ to 3 ₅, and a display device 4 are connected via anetwork.

To the AR server 1, an image of a facility in a factory (a shaft furnacein the example of FIG. 1) imaged by the camera 2 is input from thecamera 2. Note that in the following embodiments including thisembodiment, the description will be given taking examples of facilitiesin a steel plant as examples of facilities in a factory. Note that thiswill not limit the following embodiments including this embodiment.

Further, to the AR server 1, measurement information (for example,temperature information of the inside of the shaft furnace, or the like)measured by the sensors 3 ₁ to 3 ₅ is input from the sensors 3 ₁ to 3 ₅.Although five sensors are illustrated in FIG. 1, any number of sensorsmay be used. Note that in this embodiment, although the sensors 3 areprovided outside and inside the facility, they may be provided, forexample, only inside the facility or only outside the facility dependingon the facility. For example, in the case of the shaft furnace, thesensors 3 are provided inside and outside the shaft furnace. In the caseof a converter which will be described later, the sensors 3 are providedinside the converter. In the case of a continuous-casting machine whichwill be described later, the sensors 3 are provided outside thecontinuous-casting machine. The sensors are not limited to temperaturesensors measuring temperature information, and may be, for example, airpressure sensors measuring air pressure information, pressure sensorsmeasuring pressure information, gas sensors measuring the type andconcentration of gas, or the like. The sensors 3 ₁ to 3 ₅ may all be thesame type of sensors (for example, temperature sensors or the like), orthey may be different types of sensors (for example, a temperaturesensor, a gas sensor, a pressure sensor, and so on).

Note that in the following, for brevity of explanation, these pluralsensors are simply referred to as sensors unless otherwise noted.

The AR server 1 performs control to create a virtual image representingthe status of an outside or inside of the facility based on the inputmeasurement information, and overlay and display the created virtualimage and the imaged image input, from the camera 2 on the displaydevice 4.

An observer of the factory can comprehend with realistic feelings theoperating status of a facility located in an area which is dangerous andoff-limits, the operational status of the inside of a facility, or thelike by observing the display device 4.

FIG. 2 is a diagram illustrating an example of a hardware structure ofthe AR server 1.

As illustrated in FIG. 2, the AR server 1 includes a CPU 11, a storagedevice 12, and a communication device 13 as a hardware structure. TheCPU 11 executes processing or the like for controlling the AR server 1based on a program stored in the storage device 12. The storage device12 stores data and so on used by the program and the CPU 11 whenexecuting processing. The communication device 13 controls communicationbetween the AR server 1 and other devices (for example, the sensors 3,the camera 2, the display device 4, and so on).

By the CPU 11 executing processing based on the program, softwarestructures and processing related to flowcharts which will be describedlater are achieved.

FIG. 3 is a diagram illustrating an example of a software structure ofthe AR server 1 of Embodiment 1.

As illustrated in FIG. 3, the AR server 1 includes an imaged image inputunit 21, a measurement information input unit 22, a creation unit 23,and a display control unit 24 as a software structure.

The imaged image input unit 21 receives from the camera 2 an imagedimage of the facility in the factory which is imaged by the camera 2 andinputs the imaged image to the display control unit 24.

The measurement information input unit 22 receives from the sensors 3 ₁to 3 ₅ measurement information measured by the sensors 3 ₁ to 3 ₅ andinputs the measurement information to the creation unit 23.

The creation unit 23 creates a virtual image representing the status ofthe outside or inside of the facility based on the measurementinformation input by the measurement information input unit 22.

For example, the creation unit 23 receives the measurement informationfrom the measurement information input unit 22, and receives facilityinformation indicating which facility this measurement information ismeasured from the measurement information input unit 22. The creationunit 23 determines whether or not the facility indicated by the facilityinformation is a facility for which it is set to generate a virtualimage of the inside of the facility based on the received facilityinformation. When the facility indicated by the facility information isa facility for which it is set to generate a virtual image of the insideof the facility, the creation unit 23 selects a facility insideestimation model (facility inside estimation mathematical modelexpression) stored in association with the facility information from thestorage device 12 or the like based on the facility information. Morespecifically, in the storage device 12, for example, a shaft furnace anda shaft furnace inside estimation model may be stored in association.Further, in the storage device 12 or the like, for example, a converterand a converter inside estimation model may be stored in association.When the facility indicated by the facility information indicates ashaft furnace, the creation unit 23 selects the shaft furnace insideestimation model associated with the shaft furnace from the storagedevice 12 or the like. Then the creation unit 23 substitutes themeasurement information input by the measurement information input unit22 in the selected shaft furnace inside estimation model to estimate thestatus inside the shaft furnace, and creates a virtual imagerepresenting the status inside the shaft furnace.

Note that each estimation model is stored in the storage device 12merely as a logical model, and is recalculated according to, forexample, the measurement information measured by the sensors 3 ₁ to 3 ₅,to thereby generate a virtual image in a form close to the actual state.Therefore, it is preferred to be a mathematical model expression.However, it may be generated as a computer generated image in advancebased on the mathematical model expression, and the display of the imagemay be changed by changing parameters. More specifically, an adjustmentin a height direction is made according to the amount of charged rawmaterial in the shaft furnace, or in an example of a converter whichwill be described later, it is allowed to change the surface positionand so on of molten steel in the furnace based on measurement values ofsensors. Alternatively, temperature information measured by the sensors3 ₁ to 3 ₅ is applied to the shaft furnace inside estimation model togenerate an actual temperature distribution as a virtual image.

The display control unit 24 performs control to overlay and display onthe display device 4 the virtual image representing the status of theoutside or inside of the facility created in the creation unit 23 andthe imaged image of the facility input by the imaged image input unit 21(see FIG. 1 to FIG. 6).

FIG. 4 is a diagram (1) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 1. FIG. 4 illustrates anexample of overlaying and displaying an imaged image of a shaft furnaceand a virtual image of an inside of the shaft furnace created based ontemperature information of the inside of the shaft furnace.

FIG. 5 is a diagram (2) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 1. FIG. 5 illustrates anexample of overlaying and displaying an imaged image of a converter anda virtual image of an inside of the converter created based ontemperature information of the inside of the converter.

FIG. 6 is a diagram (3) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 1. FIG. 6 illustrates anexample of overlaying and displaying an imaged image of acontinuous-casting machine and steel in a molten state passing throughthe continuous-casting machine, and a virtual image in which the steelin a molten state passing through the continuous-casting machine issolidifying, which is created based on temperature information of theoutside of the continuous-casting machine.

FIG. 7 is a flowchart illustrating an example of display controlprocessing in the AR server 1 of Embodiment 1.

In step S10, the imaged image input, unit 21 receives from the camera 2an imaged image of the facility in the factory which is imaged by thecamera 2 and inputs the imaged image to the display control unit 24.

In step S11, the measurement information input unit 22 receives from thesensors 3 ₁ to 3 ₅ the measurement information measured by the sensors 3₁ to 3 ₅ and inputs the measurement information to the creation unit 23.

In step S12, the creation, unit 23 creates a virtual image representingthe status of the outside or inside of the facility based on themeasurement information input in step S11.

In step S13, the display control unit 24 performs control to overlay anddisplay on the display device 4 the virtual image representing thestatus of the outside or inside of the facility created in step S12 andthe imaged image of the facility input in step S10.

In step S14, for example, the display control unit 24 determines whetherto end the processing illustrated in FIG. 7 or not based on whetherinformation of ending display or the like is received or not from thedisplay device 4 or the like. The display control unit 24 ends theprocessing illustrated in FIG. 7 when it determines to end theprocessing, or returns the processing to step S10 when it determines notto end the processing.

Note that the imaged image input unit 21 may be configured to constantlyreceive the imaged image from the camera 2 and input the imaged image tothe display control unit 24 as long as imaging is performed by thecamera 2. Similarly, the measurement information input unit 22 may beconfigured to constantly receive the measurement information from thesensors 3 ₁ to 3 ₅ and input the measurement information to the creationunit 23 as long as measurement is performed by the sensors 3 ₁ to 3 ₅.The same applies to the following embodiments. In such a structure, theAR server 1 can perform control, to create a virtual image based on themeasurement information input in real time and overlay and display thevirtual image on the imaged image input in real time on the displaydevice 4.

As described above, according to this embodiment, the operating statusof a facility or the lake can be communicated in a realistic manner toan observer in an installation or a factory. In particular, instead ofJust showing video images, as has been conventionally performed, it ispossible to show the status of a facility which is in an operating stateat the present moment, and hence it is possible to provide anobservation in a realistic manner, which has not been present.

Embodiment 2

FIG. 8 is a diagram illustrating a system structure of a facilityguidance system of Embodiment 2. As illustrated in FIG. 8, the systemstructure of the facility guidance system of Embodiment 2 newly includesan operation server 5 compared to the system structure of the facilityguidance system of Embodiment 1.

The operation server 5 transmits control information to a facilityinside a factory. The facility executes processing based on the controlinformation. For example, when the facility is a shaft furnace, theoperation server 5 transmits to the shaft furnace charging instructioninformation of sintered ore or the like for instructing charging ofsintered ore or the like, cokes charging instruction, information forinstructing charging of cokes as an example of material, blowinginstruction information of reducing agent for instructing blowing in ofa reducing agent, tapping instruction information for instructingtapping from a tap hole, or the like.

Note that the control signal is not limited to them, and in the casewhere the facility is a converter, there may be oxygen blowinginstruction information for instructing blowing of oxygen from above theconverter, and respective blowing instruction information forinstructing blowing of oxygen or fuel gas, carbon dioxide gas, inertgas, or the like from below the furnace, and the like. Moreover, for anoperation of charging iron scraps by tilting the converter body or oftilting the converter body again to move molten steel to a ladle afterblowing, there may be instruction information for instructing tilting ofthe converter body, and the like. Further, when the facility is acontinuous-casting machine, there may be pressure instructioninformation for increasing or decreasing pressures, and the like.

To the AR server 1, the control information transmitted from theoperation server 5 to the facility is input. Then the AR server 1performs control to create a virtual image representing the status ofthe outside or inside of the facility based on the input controlinformation and the input measurement information, and overlay anddisplay the created virtual image and the imaged image input from thecamera 2 on the display device 4.

An observer of the factory can comprehend in a realistic manner theoperating status of a facility located in an area which is dangerous andoff-limits, the operational status of the inside of the facility, and soon by observing the display device 4.

FIG. 9 is a diagram illustrating an example of a software structure ofthe AR server 1 of Embodiment 2.

As illustrated in FIG. 9, the software structure of the AR server 1 ofEmbodiment 2 newly includes a control information input unit 25 comparedto the software structure of the AR server 1 of Embodiment 1.

The control information input unit 25 receives from the operation server5 the control information transmitted from the operation server 5 to thefacility and inputs the control information to the creation unit 2.

The creation unit 23 creates a virtual image representing the status ofthe outside or inside of the facility based on the measurementinformation input by the measurement information input unit 22 and thecontrol information input by the control information input unit 25. Forexample, the creation unit 23 substitutes the measurement information inthe facility inside estimation model corresponding to the facility toestimate the status inside the furnace, and creates a virtual imageaccording to the estimation result, as illustrated in Embodiment 1.

Along with this, for example when the control information is the cokescharging instruction information, the creation unit 23 adds an objectrepresenting how cokes are charged into the facility to the virtualimage based on the control information, thereby making a virtual imagerepresenting the final status inside the furnace.

FIG. 10 is a diagram (1) illustrating an example of overlay display of avirtual image and an imaged image of Embodiment 2, FIG. 10 illustratesan example of overlay and display an imaged image of the shaft furnaceand a virtual image of the inside of the shaft furnace and cokes beingcharged into the shaft furnace which is created based on temperatureinformation of the inside of the shaft furnace and the charginginstruction information of cokes into the shaft furnace.

According to this embodiment, the creation unit 23 creates the virtualimage to which not only the measurement information is added but alsothe control information is added, and thus a virtual image having morerealistic feelings can be created. Particularly in the converter, whenthe converter body is tilted, the overlay-displayed virtual imagechanges accompanying the movement, and thus the realistic feeling of theimage is increased even further.

FIG. 11 is a flowchart illustrating an example of display controlprocessing in the AR server 1 of Embodiment 2.

In step S20, the imaged image input unit 21 receives from the camera 2an imaged image of the facility in the factory which is imaged by thecamera 2 and inputs the imaged image to the display control unit 24.

In step S21, the measurement information input unit 22 receives from thesensors 3 ₁ to 3 ₅ the measurement information measured by the sensors 3₁ to 3 ₅ and inputs the measurement information to the creation unit 23.

In step S22, the control information input, unit 25 receives the controlinformation from the operation server 5 and inputs the controlinformation to the creation unit 23. Note that it is assumed that thecontrol information input unit 25 inputs the control information everytime the control information is transmitted from the operation server 5to the facility.

In step S23, the creation unit 23 creates a virtual image representingthe status of the outside or inside of the facility based on themeasurement information input in step S21 and the control informationinput in step S22.

In step S24, the display control unit 24 performs control to overlay anddisplay on the display device 4 the virtual image representing thestatus of the outside or inside of the facility created in step S23 andthe imaged image of the facility input in step S20.

In step S25, for example, the display control unit 24 determines whetherto end the processing illustrated in FIG. 11 or not based on whetherinformation of ending display or the like is received or not from thedisplay device 4 or the like. The display control unit 24 ends theprocessing illustrated in FIG. 11 when it determines to end theprocessing, or returns the processing to step S20 when it determines notto end the processing.

As described above, according to this embodiment, the operating statusof a facility or the like can be communicated in a more realistic mannerto an observer in an installation or a factory.

Embodiment 3

Depending on the factory, facilities may be in operation throughout theyear, or there may be cases where facilities are not in operation formaintenance, cleaning, or the like. In this embodiment, there will bedescribed a method for communicating, even when a facility is not inoperation, the operating status of the facility or the like in arealistic manner to an observer in an installation or a factory. Notethat a system structure, a software structure, and so on in thisembodiment are similar to those of Embodiment 1 unless otherwise noted.

The creation unit 23 of this embodiment determines whether the facilityis in operation or not based on the measurement information input fromthe measurement information input unit 22, and creates a virtual imagebased on the measurement information when it determines that thefacility is in operation, or obtains from the storage device 12 or thelike a virtual image generated in advance based on logical values oraverage values or the like of measurement information in the past whenit determines that the facility is not in operation.

For example, the creation unit 23 receives the measurement informationfrom the measurement information input unit 22, and receives facilityinformation indicating which facility this measurement information ismeasured from the measurement information input unit 22 or the like.Based on the received facility information, the creation unit 23 obtainsa virtual image generated in advance and associated with the facilityinformation from the storage device 12 or the like.

FIG. 12 is a flowchart illustrating an example of display controlprocessing in the AR server 1 of Embodiment 3.

In step S30, the imaged image input unit 21 receives from the camera 2an imaged image of the facility in the factory which is imaged by thecamera 2 and inputs the imaged image to the display control unit 24.

In step S31, the measurement information input unit 22 receives from thesensors 3 ₁ to 3 ₅ the measurement information measured by the sensors 3₁ to 3 ₅ and inputs the measurement information to the creation unit 2.

In step S32, the creation unit 23 determines whether the facility is inoperation or not based on the measurement information input from themeasurement information input unit 22. The creation unit 23 determinesthat the facility is not in operation when, for example, the measurementinformation is not in a predetermined range. The creation unit 23advances the processing to step S33 when it determines that the facilityis in operation, or advances the processing to step S34 when itdetermines that the facility is not in operation.

In step S33, the creation unit 23 creates a virtual image representingthe status of the outside or inside of the facility based on themeasurement information input in step S31.

On the other hand, in step S34, the creation unit 23 obtains from thestorage device 12 or the like a virtual image representing the status ofthe outside or inside of the facility, which is created in advance basedon logical values or the like corresponding to the facility.

In step S35, the display control unit 24 performs control to overlay anddisplay on the display device 4 the virtual image representing thestatus of the outside or inside of the facility created in step S33 orobtained in step S34 and the imaged image of the facility input in stepS30.

In step S36, for example, the display control unit 24 determines whetherto end the processing illustrated in FIG. 12 or not based on whetherinformation of ending display or the like is received or not from thedisplay device 4 or the like. The display control unit 24 ends theprocessing illustrated in FIG. 12 when it determines to end theprocessing, or returns the processing to step S30 when it determines notto end the processing.

As described above, according to this embodiment, when the facility isin operation, a virtual image created based on the measurementinformation is overlaid and displayed on an imaged image of thefacility, and thereby the operating status of a facility or the like canbe communicated with realistic feelings to an observer in aninstallation or a factory. Further, according to this embodiment, whenthe facility is not in operation, a virtual image created based onlogical values or average values of measurement information in the pastis overlaid and displayed on an imaged image of the facility, andthereby the operating status of a facility or the like can becommunicated in a realistic manner to an observer in an installation ora factory.

Note that the system structure, the software structure, and so on ofthis embodiment are made similar to those in Embodiment 1, but when theyare made similar to those in Embodiment 2, the creation unit 23 may beconfigured to determine whether the facility is in operation or notbased on the measurement information and/or the control information. Forexample, the creation unit 23 may be configured to determine that thefacility is not in operation when the measurement information is not ina predetermined range and/or when the control information is nottransmitted for a predetermined period from, the operation server 5 tothe facility.

Embodiment 4

The display control unit 24 may be configured to transmit a displaylayer set for each facility to the display device 4 in response to ascreen operation or the like by an observer on the display device 4.FIG. 13A to FIG. 13D are diagrams illustrating an example of displayingthe facility by each display layer. FIG. 13A represents an externalappearance of a shaft furnace. FIG. 13B represents fireproof bricksinside the shaft furnace. FIG. 13C represents appearances of iron oreand cokes in the shaft furnace. FIG. 13D represents a cross section ofthe shaft furnace.

When display of a display layer similar to that in FIG. 13A or FIG. 13Bis requested from the display device 4, the display control unit 24performs control to obtain a virtual image of the display layer createdin advance from the storage device 12 or the like, and overlay anddisplay on the display device 4 the obtained virtual image of thedisplay layer and the imaged image input by the imaged image input unit21.

According to this embodiment, a virtual image displaying the operatingstatus of a facility or the like with various types of display layers isoverlaid and displayed on an imaged image of the facility in response toa request or the like from an observer, and thereby the operating statusof a facility or the like can be communicated with more realisticfeelings and in detail to an observer in an installation or a factory.

Embodiment 5

FIG. 14 is a diagram illustrating an example of a system structure of afacility guidance system of Embodiment 5. As illustrated in FIG. 14, thesystem structure of the facility guidance system of Embodiment 5 doesnot include the camera 2 compared to the system structure of thefacility guidance system of Embodiment 1. Further, the display device 4of this embodiment is a transmissive liquid crystal film fixed to awindow or the like for example.

To the AR server 1, measurement information (for example, temperatureinformation of the inside of a shaft furnace, or the like) measured bythe sensors 3 ₁ to 3 ₅ is input from the sensors 3 ₁ to 3 ₅.

The AR server 1 performs control to create a virtual image representingthe status of the outside or inside of the facility based on the inputmeasurement information, and overlay and display the created virtualimage on a facility seen through the window or the like on which thedisplay device 4 is provided.

FIG. 15 is a diagram illustrating an example of a software structure ofthe AR server 1 of Embodiment 5.

As illustrated in FIG. 15, the AR server 1 includes a measurementinformation input unit 22, a creation unit 23, and a display controlunit 24 as a software structure.

The measurement information input unit 22 receives from the sensors 3 ₁to 3 ₅ measurement information measured by the sensors 3 ₁ to 3 ₅ andinputs the measurement information to the creation unit 23.

The creation unit 23 creates a virtual image representing the status ofthe outside or inside of the facility based on the measurementinformation input by the measurement information input unit 22.

The display control unit 24 performs control to overlay and display thevirtual image representing the status of the outside or inside of thefacility which is created in the creation unit 23 on the facility seenthrough the window or the like on which the display device 4 isprovided.

FIG. 16 is a flowchart illustrating an example of display controlprocessing in the AR server 1 of Embodiment 5.

In step S40, the measurement information input unit 22 receives from thesensors 3 ₁ to 3 ₅ the measurement information measured by the sensors 3₁ to 3 ₅ and inputs the measurement information to the creation unit 23.

In step S41, the creation unit 23 creates a virtual image representingthe status of the outside or inside of the facility based on themeasurement information input in step S40.

In step S42, the display control unit 24 performs control to overlay anddisplay the virtual image representing the status of the outside orinside of the facility which is created in step S41 on the facility seenthrough the window or the like on which the display device 4 isprovided.

In step S43, for example, the display control unit 24 determines whetherto end the processing illustrated in FIG. 16 or not based on whetherinformation of ending display or the like is received or not from thedisplay device 4 or the like. The display control unit 24 ends theprocessing illustrated in FIG. 16 when it determines to end theprocessing, or returns the processing to step S40 when it determines notto end the processing.

As described above, also in this embodiment, the operating status of afacility or the like can be communicated in a realistic manner to anobserver in an installation or a factory.

Embodiment 6

Even now, when computerized control is growing, control of factoryfacilities in a steel plant is often performed such that an operatorpredicts the status of the inside of a shaft furnace or converter fromthe color of pig iron seen through a viewing window of the shaftfurnace, flames bursting up through the converter opening, or the like,and the operator performs control based on a decision made by theoperator according to the status at the time. However, this decision isnot easy since it requires being skillful, and when the decision by theoperator is not appropriate, it may result in causing deterioration ofthe quality of products produced through the factory facilities.

Accordingly, in this embodiment, there will be described an operationsupport system which simulates, when control, of a factory facility isperformed by the operator's decision, how the status of the factoryfacility (for example, the status of the inside of a shaft furnace orconverter) would be when this control is performed, and presents resultsof the simulation as a three-dimensional image to the operator.

In the facility guidance system of Embodiments 1 to 5, athree-dimensional virtual image representing the current status of theinside of a facility is created based on measurement information andcontrol information, and this image is presented to the observer.However, based on information (control schedule information)representing the contents at control to be performed from the presentmoment besides the measurement information, the operation support systemaccording to Embodiment 6 predicts by simulation how the status of theinside of the facility would change if this control is performed, andpresents prediction results to the operator. Considering the predictionresults, the operator finally determines what control should be actuallyperformed.

More specifically, the operation support system according to thisembodiment is as follows. FIG. 17 is a diagram illustrating an exampleof a system structure of the operation support system of Embodiment 6.As illustrated in FIG. 17, in the operation support system, a server101, an HMD 102 with camera (hereinafter abbreviated as HMD 102), andsensors 103 ₁ to 103 ₅ are connected via a wireless network.

FIG. 18 is a diagram illustrating an example of a software structure ofthe operation support system of Embodiment 6.

The software structure of the operation support system of thisembodiment newly includes a control schedule information input unit 26as compared to the software structure of the AR server 1 of Embodiment1.

Further, in this embodiment, functions of software are distributed amongthe server 101 and the HMD 102. The server 101 includes a measurementinformation input unit 22, a control schedule information input unit 26,and a simulation unit 27 as a functional structure of software. Further,the HMD 102 has a camera and a drawing function, and includes an imagedimage input unit 21 and a drawing/display control unit 28 as afunctional structure of software. Note that in this embodiment althoughthe functions of software are distributed among the server 101 and theHMD 102, it may be structured such that the server 101 includes all thefunctions as in Embodiment 1 or the like, or may be structured such thatthe HMD 102 includes all the functions, and hence it is not limited inparticular.

First, the function of the server 101 will be described.

The measurement information input unit 22 receives measurementinformation measured by the sensors 103 ₁ to 103 ₅ as input data fromthe sensors 103 ₁ to 103 ₅. At this moment, the sensors 103 ₁ to 103 ₅transmit the measurement information to the server 101 via the wirelessnetwork.

The control schedule information input unit 26 receives control scheduleinformation input by the operator as input data. As the control scheduleinformation, similarly to the control information, there are charginginstruction information of sintered ore or the like, cokes charginginstruction information, blowing instruction information of reducingagent, and tapping instruction information for instructing tapping froma tap hole, as well as blowing instruction of oxygen, air, or varioustypes of gas and pressure instruction information. The operator caninput the control schedule information via an input device connected tothe server, or can input the control schedule information remotely via aportable terminal or the like.

Based on the measurement information received in the measurementinformation input unit 22 and the control schedule information receivedin the control schedule information input unit 26, the simulation unit27 simulates the status of the outside or inside of the facility whencontrol for which contents are represented by the control scheduleinformation is performed. For example, the simulation unit 27substitutes the measurement information (current temperature of theshaft furnace, or the like) and control schedule information (blowinginstruction information representing the amount of blowing oxygen or airvia each tuyere, or the like) in the facility inside estimation modelcorresponding to the facility, performs numerical analysis on the statusin the facility, and outputs numeric data representing results thereof.

Note that as a technique using the facility inside estimation model, forexample, Japanese Laid-open Patent Publication No. 8-295910 or the likediscloses a technique to simulate operation of a shaft furnace byintroducing a mathematical expression model modeling the state in thefurnace. Using such a technique, data of temperatures, pressures, and soon at each point of three-dimensional coordinates in the furnace can beobtained.

Next, the function of the HMD 102 will be described.

The imaged image input unit 21 receives as input data an imaged image ofthe facility in the factory imaged by the camera provided in the HMD102.

Upon reception of numeric data representing simulation results by thesimulation unit 27, the drawing/display control unit 28 draws athree-dimensional virtual image based on the numeric data. Moreover, thedrawing/display control unit 28 performs control to overlay and displayon the display device of the HMD 102 a three-dimensional virtual imagerepresenting simulation results of the status of the outside or insideof the drawn facility and the imaged image of the facility input by theimaged image input unit 21. At this time, the server 101 transmitsnumeric data representing simulation results to the HMD 102 via thewireless network.

FIG. 19 is a flowchart illustrating an example of processing in theoperation support system of Embodiment 6.

In step S50, the imaged image input unit 21 receives as input data animaged image of the facility in the factory imaged by the cameraprovided in the HMD 102.

In step S51, the measurement, information input unit 22 receivesmeasurement information measured by the sensors 103 ₁ to 103 ₅ as inputdata from the sensors 103 ₁ to 103 ₅.

In step S52, the control schedule information input unit 26 receivescontrol schedule information input by the operator.

In step S53, based on the measurement information input in step S51 andthe control schedule information input in step S52, the simulation unit27 executes simulation estimating the status of the outside or inside ofthe facility, and outputs numeric data representing simulation results.Then, the drawing/display control unit 28 creates a three-dimensionalvirtual image based on the numeric data representing the simulationresults.

In step S54, the drawing/display control unit 28 performs control tooverlay and display on the display (display device) provided in the HMD102 the three-dimensional virtual image representing the simulationresults of the status of the outside or inside of the facility createdin step S53 and the imaged image of the facility input in the step S50.

In step S55, for example, the drawing/display control unit 28 determineswhether to end the processing illustrated in FIG. 19 or not based onwhether information of ending display or the like is received or notfrom the HMD 103 or the like. The drawing/display control unit 28 endsthe processing illustrated in FIG. 19 when it determines to end theprocessing, or returns the processing step to step S50 when itdetermines not to end the processing.

Regarding the processing of overlay-displaying the three-dimensionalvirtual image created by the drawing/display control unit 28 and theimaged image, a more detailed description will be given below, taking anexample of using blowing instruction information representing the amountof oxygen or air being blown in via each tuyere according to the controlschedule information.

In a large shaft furnace, tuyeres for sending air into the furnace areprovided at 30 to 40 positions in a lower part of the main body thereof,where the temperature distribution in the periphery of each tuyerediffers when the amount of sending into the tuyere differs.

Accordingly, each tuyere is assigned an ID for identification(hereinafter abbreviated as a tuyere ID), and when the simulation unit27 executes simulation, blowing instruction information representing theamount of oxygen or air to blown is input with a specified tuyere ID asthe control schedule information. Thus, there is input data representingthe amount of blowing at each point in the three dimensional coordinatescorresponding to each tuyere. Further, as initial values of temperaturedata at each point in the three-dimensional coordinates in the shaftfurnace, there are input values calculated based on the measurementinformation from the sensors.

Then, the simulation unit 27 performs numerical analysis of the equationof motion of flowing and/or heat conduction equation based on amathematical expression model representing the state in the shaftfurnace with the input control schedule information and measurementinformation being initial conditions, to thereby calculate variousnumeric data (temperature data, velocity data, pressure data, and so on)at each point in the three-dimensional coordinates in the shaft furnaceafter a certain time has passed.

Then, upon reception of the numeric data at each point of thethree-dimensional coordinates in the shaft furnace which representsimulation results from the simulation unit 27, the drawing/displaycontrol unit 28 generates a three-dimensional virtual image indicatingthe simulation results (temperature distribution, velocity distribution,pressure distribution, and so on) from the received numeric data.

Moreover, when the drawing display control unit 28 overlay-displays thecreated three-dimensional virtual image and the imaged image, forexample, the drawing/display control unit identifies the direction andangle of a portion which is seen by the user based on the image of theshaft furnace imaged by the camera provided in the HMD 102. Morespecifically, the image of the shaft furnace imaged by the cameraprovided in the HMD 102 is substantially the same as the image of theshaft furnace captured in the visual field of the user, and thus, withimages obtained by imaging the shaft furnace in advance from variousangles and directions being stored in a memory in the HMD 102 inadvance, the direction and the angle of the portion seen by the user isidentified by matching the image imaged by the camera of the HMD 102 andthe images stored in the memory. Then, based on the identified directionand angle, the drawing/display control unit 28 controls the direction ofthe three-dimensional virtual image representing the simulation resultsto be presented to the operator.

Note that it may be structured to allow the operator to adjust thedirection of the three-dimensional virtual image by himself/herself viaan input device, and hence it is not particularly limited.

This structure allows the operator to recognize in advance, from thesimulation results, what status the temperature distribution and thepressure distribution in the shaft furnace or the converter would be dueto control to be currently performed by him/her.

Thus, for example, the operator is able to perform scheduled controlwhen the results of simulation inside the shaft furnace indicate afavorable temperature distribution and so on, or reconsider the contentsof control and select an appropriate operation when the results ofsimulation indicate an unfavorable temperature distribution and so on.Therefore, even a less-experienced operator is able to appropriatelyoperate a factory facility such as the shaft furnace, which requiresbeing skillful. Furthermore, since the inside temperature distributionand so on are presented as a three-dimensional virtual image which isoverlaid on an actual shaft furnace or converter, the operator is ableto comprehend the status inside the facility more intuitively, and isable to instantly and accurately comprehend whether the results ofsimulation are favorable or not.

Note that in this embodiment, although the case of using the no HMD 102with camera has been described as an example, it may be of a structureincluding no camera by applying the structure of Embodiment 5 and usinga transmissive HMD. That is, the HMD does not include the imaged imageinput unit 21, and the drawing/display control unit 28 performs controlto overlay and display the created three-dimensional virtual image on afacility which is actually seen through the HMD. In this case, when theHMD includes a GPS and an acceleration sensor, the HMD is able torecognize from what direction and angle the user is capturing thefacility. Further, the relation between the three-dimensionalcoordinates used in the simulation and the actual direction of thefacility is defined in advance. Thus, even in the HMD having no camera,it is possible to control the direction of the three-dimensional virtualimage representing simulation results to be presented to the operatorbased on the identified direction and angle, and overlay and display theimage on the facility seen through the transmissive HMD.

Embodiment 7

In a steel, plant, molten iron coming out of a shaft furnace is put in atorpedo car or the like and carried to a steel factory, and charged intoa steel furnace in a molten state as it is. Then, assuming that thetorpedo car is also a part of factory facilities, a virtual imagerepresenting the status inside may be generated and overlay-displayed onthe torpedo car to be presented to a worker, similarly to Embodiment 1.Specifically, it may be structured such that measurement information oftemperatures and pressures of the inside or surface of the torpedo car,and so on is obtained by sensors, and a virtual image representing atemperature distribution and so on generated based on a facility insideestimation model associated with the torpedo car using this measurementinformation is overlaid and displayed on the actual torpedo car.

Moreover, in this embodiment, it may be structured to display the ordernumber corresponding to molten iron carried by the torpedo car.Describing in more detail, the operation support system according tothis embodiment is as follows. FIG. 20 is a diagram illustrating anexample of a system structure of the operation support system ofEmbodiment 7. As illustrated in FIG. 20, in the operation supportsystem, a server 101, an HMD 102 with camera (hereinafter abbreviated asHMD 102), and sensors 104 ₁ to 104 ₅ are connected via a wirelessnetwork.

Note that in Embodiment 7, the various types of processing performed bythe AR server 1 in Embodiment 1 are performed in the HMD 102. Describingin more detail, the function of the HMD 102 is as follows.

The HMD 102 obtains an imaged image of a facility (torpedo car in theexample of FIG. 20) in a factory which is imaged by the camera providedintegrally in the HMD. Further, the HMD 102 obtains measurementinformation (for example, temperature information of the inside of thetorpedo car, or the like) measured by the sensors 104 ₁ to 104 ₅ viawireless communication or the like.

Moreover, in this embodiment, a marker for identifying a facility isadded to each facility in the factory, and the HMD 102 is capable ofidentifying each facility by the camera provided in the HMD recognizingeach marker. More specifically, the HMD 102 retains in a memory providedin the HMD a facility marker correspondence table illustrated in FIG. 21in which facility IDs for identifying respective facilities andrespective markers are correlated, and identifies each facility byrecognizing a marker by the camera and reading the facility IDcorrelated with the marker from the facility marker correspondencetable. FIG. 21 is a diagram illustrating an example of the facilitymarker correspondence table.

Further, the HMD 102 creates a virtual image representing the status ofthe outside or inside of the facility based on the obtained measurementinformation, and performs control to overlay and display the createdvirtual image and the imaged image imaged by the camera on a display(display device) provided in the HMD.

Moreover, this embodiment has a characteristic in that the HMD 102generates a virtual image indicating an order number assigned to aproduct in the facility based on the facility ID, and overlays anddisplays on the display device provided in the HMD the created virtualimage of the order number and the imaged image of the facility imaged bythe camera. Describing more specifically, this point is as follows.

In this embodiment, the server 101 stores a facility usage status tableillustrated in FIG. 22 in which facility IDs for identifying respectivefacilities and order numbers are correlated. This facility usage statustable is updated by the operator when, for example, data is registeredin a production management system in the factory. For example, when anoperation to produce pig iron in a shaft furnace is started, the ordernumber associated with the pig iron to be produced from the presentmoment in the shaft furnace is input by the operator to the productionmanagement system, thereby updating the facility usage status table.That is, in this embodiment, the server 101 stores data of theproduction management system, which is different from the AR server 1illustrated in FIG. 2. FIG. 22 is a diagram illustrating an example ofthe facility usage status table.

Note that the hardware structure of the HMD 102 includes, similarly tothe AR server 1 illustrated in FIG. 2, a CPU, a storage device (memory),and a communication device, and the CPU executes processing ofcontrolling the HMD 102 based on a program and data included in thestorage device, or the like, thereby achieving a software structure andprocessing according to a flowchart which will be described later.

FIG. 23 is a diagram illustrating an example of a software structure ofthe HMD with camera. As illustrated in FIG. 23, the HMD 102 includes animaged image input unit 21, a measurement information input unit 22, anidentification processing unit 29, a creation unit 23, and a displaycontrol unit 24 as a software structure.

The imaged image input unit 21 receives as input data an imaged image ofa facility in a factory imaged by the camera provided in the HMD 102.

The measurement information input unit 22 receives measurementinformation measured by the sensors 3 ₁ to 3 ₅ as input data from thesensors 3 ₁ to 3 ₅.

The identification processing unit 29 identifies an order numberassigned to a product in each facility. More specifically, theidentification processing unit 29 reads the facility ID corresponding tothe marker added to the facility in the factory which is captured by thecamera from the above-described facility marker correspondence table.

In the example illustrated in FIG. 20, when the camera provided in theHMD captures a marker A added to a torpedo car A and a marker B added toa torpedo car B, the identification processing unit 29 reads facility IDcorresponding to each marker from the facility marker correspondencetable, and identifies that the facility to which the marker. A is addedis the torpedo car A, and that the facility to which the marker B isadded is the torpedo car B.

Moreover, the identification processing unit 29 reads the order numbercorresponding to each facility ID from the above-described facilityusage status table, and identifies the order number assigned to aproduct or the like in each facility.

In the example illustrated in FIG. 20, when the identificationprocessing unit 29 identifies the torpedo car A and the torpedo car B,the identification processing unit reads the order IDs corresponding torespective facility IDs from the facility usage status table, andidentifies that the pig irons of the order IDs “ABC123”, “EDF456” arecarried in the torpedo car A, and pig irons of the order IDs “G123”,“X456” are carried in the torpedo car B.

The creation unit 23 creates a virtual image representing the status ofthe outside or inside of the facility based on the measurementinformation input by the measurement information input unit 22. Further,the creation unit 23 creates a virtual image representing the ordernumbers identified in the identification processing unit 29.

The display control unit 24 performs control to overlay and display onthe display device of the HMD 102 the virtual image representing thestatus of the outside or inside of the facility which is created in thecreation unit 23 and the imaged image of the facility input by theimaged image input, unit 21. Further, the display control unit 24performs control to overlay and display on the display device of the HMD102 the virtual image representing the order numbers which is created inthe creation unit 23 and the imaged image of the facility input by theimaged image input unit 21.

FIG. 20 illustrates how the virtual image representing the order numberswhich is created in the creation unit 23 is overlaid by the displaycontrol, unit 24 on the image of the torpedo cars imaged by the camerain the HMD 102.

Note that although. FIG. 20 does not illustrate an example of overlayingthe virtual image representing the status of the outside or inside ofthe facility, it can be displayed by switching display layers using thestructure of Embodiment 4.

Further, in the example illustrated in FIG. 20, although the structureusing a marker for identifying a factory facility is described, awireless IC tag or the like such as RFID may be used for identifying afactory facility instead of the marker. Specifically, it may bestructured such that the HMD 102 includes a reader function for awireless IC tag, a wireless IC tag storing a facility ID is added toeach facility, and the facility ID is read directly therefrom, and henceit is not limited in particular.

FIG. 24 is a flowchart illustrating an example of processing in theoperation support system of Embodiment 7.

In step S60, the imaged image input unit 21 receives an imaged image ofa facility in a factory which is imaged by the camera provided in theHMD 102 and inputs the imaged image to the display control unit 24.

In step S61, the measurement information input unit 22 receives from thesensors 3 ₁ to 3 ₅ measurement information measured by the sensors 3 ₁to 3 ₅ and inputs the measurement information to the creation unit 23.

In step S62, the identification processing unit 29 identifies thefacility ID of each facility based on a marker captured by the cameraprovided in the HMD 102, and further identifies the order number of aproduct or the like stored in each facility based on the identifiedfacility ID.

In step S63, the creation unit 23 creates a three-dimensional virtualimage representing the status of the outside or inside of the facilitybased on the measurement information input in step S61. Further, thecreation unit 23 creates a virtual image representing an order numberbased on the order number identified in step S62.

In step S64, the display control unit 24 performs control to overlay anddisplay on the display device 4 the three-dimensional virtual imagerepresenting the status of the outside or inside of the facility whichis created in step S63 and the imaged image of the facility input instep S60. Further, the display control unit 24 performs control tooverlay and display on the display device 4 the virtual imagerepresenting the order number created in step S63 and the imaged imageof the facility input in step S60.

In step S65, for example, the display control unit 24 determines whetherto end the processing illustrated in FIG. 24 or not based on whetherinformation of ending display or the like is received or not from thedisplay device 4 or the like. The display control unit 24 ends theprocessing illustrated in FIG. 24 when it determines to end theprocessing, or returns the processing step to step S60 when itdetermines not to end the processing.

Note that this embodiment can be applied not only to facilities in afactory but also to a product itself waiting for shipment. In this case,it may be structured such that S61 is skipped, because it is unnecessaryto overlay-display a virtual image representing the appearance of theinside or outside of a facility, and the processing of generating avirtual image representing the appearance of the outside or inside ofthe facility in S63 is not performed, and hence it is not limited inparticular.

FIG. 25 is a diagram illustrating an example of the system structure ofthe operation support system of Embodiment 7. In the example illustratedin FIG. 25, an example of displaying order numbers on coils waiting forshipment is illustrated. In this example, markers 105 ₁ to 105 ₃ areadded to the coils waiting for shipment, and by capturing these markerswith the HMD 102, the order numbers of the respective coils can beobtained. However, details of processing contents are similar to theexample described using FIG. 20, and thus the description thereof isomitted.

Further, in the example illustrated in FIG. 25, not only the ordernumbers are displayed by AR, but also a loading order instruction isdisplayed by AR. In this case, the operator registers informationrelated to the order of loading (information defining the order ofloading or the like using order numbers) in advance in the server 101.Then, when the identification processing unit 29 identifies ordernumbers of products, the information related to the order of loadingregistered in the server 101 is read at the same time, and the creationunit 23 creates a virtual image representing the loading orderinstruction.

Thus, just by looking at a facility or a product in a factory, a workercan immediately comprehend an assigned order number and the order ofloading regarding products, half finished products, materials, and/orthe like stored in a facility, and thus operations such as preparationfor shipment and transportation to the next process can be performedsmoothly, and mistakes in operation can be prevented.

Note that also in this embodiment, although the case of using thenon-transmissive HMD 102 with camera has been described as an example,it may be of a structure including no camera by applying the structureof Embodiment 5 and using a transmissive HMD. In this case, the HMD doesnot include the imaged image input unit 21, and the creation unit 23performs control to overlay and display the created three-dimensionalvirtual image on a facility which is actually seen through the HMD.

In the foregoing, preferred embodiments of the present invention havebeen described in detail, but the present invention is not limited tosuch specific embodiments. Various variations and modifications may bemade within the scope of the spirit of the present invention describedin the claims.

For example, the above-described embodiments are described with examplesof showing the appearance of the inside of a facility, such as a shaftfurnace or a converter in a steel plant, the inside of which isdifficult to be seen. However, the above-described embodiments may alsobe applied to a facility handling liquid or powder in a transparentcasing in the case of food manufacturing industry, chemical orpharmaceutical manufacturing industry, or the like. When theabove-described embodiments are applied in such a facility, liquid orpowder which is being charged can be shown directly to an observer, andattributes (material name, temperature, viscosity, acidity, alcoholcontent, amount per charge, and so on of the liquid or powder can beoverlay-displayed in the form of display like a tag. In this manner,more specific information can be obtained from tag information whileobserving an operating status in real time, and thus more intelligibleobservation trip service can be provided to an observer.

Further, although the above-described embodiments are described takingexamples of what is called a non-transmissive display or a transmissivedisplay, the present invention can also be applied to a non-transmissiveprojector or a transmissive projector. For example, instead of thedisplay device 4, a display device having a display unit in an eyeglassform which is worn on an observer's head or the like may be used.Further, for example, when overlaying and displaying an imaged image anda virtual image, the AR server 1 aligns the coordinates of the imagedimage and the virtual image. As a method for aligning coordinates, forexample, there is a method to dispose a fixed marker in advance on afacility, and align the coordinates of the imaged image and the virtualimage based on the fixed marker in an image imaged by the camera 2. Notethat when the display device having the display unit in an eyeglass formwhich is worn on an observer's head or the like is used, the AR server 1presumes where the observer is looking at based on position information,direction information, and so on from the display device, and aligns thecoordinates to match the presumed position.

Note that the above-described embodiments may be combined arbitrarilyand implemented.

Note that the above-described AR server 1 is an example of a computer.

Embodiment 8

FIG. 26 is a diagram illustrating an example of a structure of aninformation providing system according to this embodiment. Theinformation providing system has an AR providing apparatus 1000 and aninformation processing apparatus 200. The AR providing apparatus 1000and the information processing apparatus 200 are connected communicablyvia a network.

The AR providing apparatus 1000 is an example of an informationproviding apparatus (computer), is an HMD (Head Mounted Display) or thelike, and provides augmented reality (AR) by displaying an image(computer graphics image) generated in the AR providing apparatus 1000at a position which matches a real space which can be perceived via theAR providing apparatus 1000. The AR providing apparatus 1000 overlaysand displays, on a crane truck 300 in a real space for example, adangerous range image 310 indicating a dangerous range due to the cranetruck 300. Note that the information processing apparatus 200 is aserver computer for example, and manages various types of informationrelated to a dangerous object (which, is an example of a dangeroustarget) such as the crane truck 300. In addition, the dangerous targetis not limited to the dangerous object itself but includes a place or aspace related to a danger, such as a road where a large, unspecifiednumber of dangerous objects pass by frequently.

FIG. 27 is a diagram illustrating an example of a hardware structure ofthe AR providing apparatus 1000.

The AR providing apparatus 1000 has a control device 1005, a storagedevice 1010, a communication device 1015, a display device 1020, adirection detection device 1025, a posture detection device 1030, and animaging device 1035.

The control device 1005 is a CPU (Central Processing Unit) for example,and reads a program from the storage device 1010 as necessary andexecutes the program. By the program being executed, there are achievedfunctions in the AR providing apparatus 1000 which will be describedlater and processing related to flowcharts which will be describedlater.

The storage device 1010 is a ROM (Read Only Memory), a RAM (RandomAccess Memory), an HD (Hard Disk), and/or the like and stores varioustypes of information. Described in more detail, the storage device 1010(ROM) stores a program and the like which are read first when the powerof the AR providing apparatus 1000 is turned on. Further, the storagedevice 1010 (RAM) functions as a main memory of the AR providingapparatus 1000. Further, the storage device 1010 (HD) stores numericdata and the like calculated by the control device 1005 other than theprogram. Note that the AR providing apparatus 1000 may obtain varioustypes of information to be stored in the storage device 1010 from arecording medium such as a CD-ROM, or may download them via a network orthe like.

The communication device 1015 performs communication via wire orwirelessly with the information processing apparatus 200 to obtainvarious types of information related to a dangerous object. Further, thecommunication device 1015 performs communication with a satellite, whichis an example of a position detection device, to obtain orbitinformation.

The display device 1020 is an example of a display unit, is atransmissive liquid crystal display or the like, and displays varioustypes of images.

The direction detection device 1025 is an electronic compass forexample, detects weak geomagnetism (for example, geomagnetism in aforward and backward direction and geomagnetism in a leftward andrightward direction), and calculates the direction (directioninformation) of the AR providing apparatus 1000 by calculating thedirection of the north from the intensity of the geomagnetism.

The posture detection device 1030 is a gyro sensor for example, detectsthe angular velocity of an object and calculates an angle (posture ofthe AR providing apparatus 1000 (posture information)) by integratingthe angular velocity, or the like.

The imaging device 1035 performs imaging of a real space.

Note that the hardware structure of the AR providing apparatus 1000 isnot limited to this. For example, a direction posture detection devicehaving a function integrating the functions of the direction detectiondevice 1025 and the posture detection device 1030 may be employedinstead of the direction detection device 1025 and the posture detectiondevice 1030.

FIG. 28 is a diagram illustrating an example of a functional structureof the AR providing apparatus 1000.

The AR providing apparatus 1000 has a measurement unit 1040, a directioninformation obtaining unit 1045, a posture information obtaining unit1050, a retrieval unit 1055, a visual field determination unit 1060, adisplay control unit 1065, and a video obtaining unit 1070.

The measurement unit 1040 is an example of a position informationobtaining unit and calculates (calculates and obtains) information (ARposition information as an example of apparatus position information)indicating the current position of the AR providing apparatus 1000 fromthe orbit information obtained from a satellite via the communicationdevice 1015.

The direction information obtaining unit 1045 obtains directioninformation calculated in the direction detection device 1025. Note thatthe direction information obtaining unit 1045 may receive detectedinformation such as geomagnetism from the direction detection device1025 and calculate the direction information.

The posture information obtaining unit 1050 obtains posture informationcalculated in the posture detection device 1030, Note that the postureinformation obtaining unit 1050 may receive detected information such asangular velocity from the posture detection device 1030 and calculateposture information.

The retrieval unit 1055 retrieves (reads) necessary information fromtables (positionally fixed target table which will be described later,mobile target table which will be described later, and so on) storinginformation, which is used when a dangerous range image indicating thedangerous range due to a dangerous object is generated, as various typesof information related to dangerous objects from a storage device 280which will be described later and is included in the informationprocessing apparatus 200.

The visual field determination unit 1060 determines (estimates) thevisual melt (AR visual field) of the user wearing the AR providingapparatus 1000 based on the direction information obtained in thedirection information obtaining unit 1045, the posture informationobtained in the posture information obtaining unit 1050, and so on.

The display control unit 1065 generates an augmented image based on thedirection information obtained in the direction information obtainingunit 1045, the posture information obtained in the posture informationobtaining unit 1050, the information retrieved in the retrieval unit1055, and so on and displays the augmented image on the display device1020. At this moment, the display control unit 1065 performs alignmentof coordinates for displaying the augmented image matched with thedangerous object in the real space on the display device 1020.

To accurately perform alignment of coordinates (geometric alignment)between the real space and the displayed space (virtual space), it isnecessary to match parameters (internal parameters and externalparameters) for generating an augmented image of a virtual space withparameters of the display device 1020. Internal parameters of thedisplay device 1020 are known, and thus the problem of geometricalignment eventuates in a problem to obtain external parameters of thedisplay device 1020 (that is, the position and, posture of the ARproviding apparatus 1000).

Here, to obtain the position and posture of the AR providing apparatus1000, it is possible to employ various methods by appropriatelycombining the direction detection device 1025 (magnetic sensor), theposture detection device 1030 (gyro sensor), the control device 1005(GPS), and the display device 1020. In other words, they can be roughlycategorized into methods using measurement results by sensors (ARposition information, direction information, and posture information),methods to use an image imaged by the display device 1020, and methodsto complement a displacement in alignment by a combination of thedisplay device 1020 and sensors.

In this embodiment, on the assumption that the internal parameters(focal distance, lens distortion, aspect ratio, and so on) of thedisplay device 1020 are calibrated in advance, the description will begiven employing a method in which the display control unit 1065 obtainsthe AR position information, the direction information, and the postureinformation of the AR providing apparatus 1000 which are measured, so asto obtain a geometric transformation matrix between a virtual coordinatesystem representing a virtual environment and a reference coordinatesystem of a real space which is given in advance (conversion informationcorrelating conversion between a virtual coordinate system and areference coordinate system).

Note that as described above, the display control unit 1065 may obtainthe geometric transformation matrix by estimating, for example, theexternal parameters (position and posture) of the display device 1020 inreal time using some kind of coordinates existing in an imaged image. Atthis time, as the index for estimating the position and posture of theAR providing apparatus 1000, an intentionally added index such as amarker may be used, or natural characteristics such as contourinformation, brightness edge and feature points may be used.

The video obtaining unit 1070 obtains video data of the real spaceimaged in the imaging device 1035.

FIG. 29 is a diagram illustrating an example of a hardware structure ofthe information processing apparatus 200.

The information processing apparatus 200 has a control device 275, astorage device 280, a communication device 235, a display device 290,and an input device 295.

The control device 275 is a CPU (Central Processing Unit) for example,and reads a program from the storage device 280 as necessary andexecutes the program. By the program being executed, there are achievedfunctions in the information processing apparatus 200 which will bedescribed later and processing related to flowcharts which will bedescribed later.

The storage device 280 is an example of a storage unit and is a ROM(Read. Only Memory), a RAM (Random Access Memory), an HD (Hard Disk),and/or the like and stores various types of information. Describing inmore detail, the storage device 280 (ROM) stores a program and the likewhich are read first when the power of the information processingapparatus 200 is turned on. Further, the storage device 280 (RAM)functions as a main memory of the information processing apparatus 200.Further, the storage device 280 (HD) stores numeric data and the likecalculated by the control device 275 other than the program. Note thatthe information processing apparatus 200 may obtain various types ofinformation to be stored in the storage device 280 from a recordingmedium such as a CD-ROM, or may download them via a network or the like.

The communication device 285 communicates with an operating statedetection sensor detecting a state that a dangerous object is operating(operating state) and obtains information (dangerous object positioninformation which is an example of dangerous target positioninformation) or the like indicating the current position of thedangerous object from the operating state detection sensor.

The display device 290 is a liquid crystal display or the like anddisplays various types of information.

The input device 295 is a keyboard and a mouse or the like operated bythe user, and inputs various types of information to the informationprocessing apparatus 200.

FIG. 30 is a diagram illustrating an example of a table (positionallyfixed target table 210) storing information related to dangerousobjects.

The positionally fixed target table 210 is structured to includeinformation of dangerous object ID, dangerous object name, type ofdanger, reference position, state, dangerous range of each state, andcurrent status. Various types of information excluding the informationof current status are set via the input device 295 or the like by anadministrator for example. The setting of the information of currentstatus will be described later.

The information of dangerous object ID is identification information foridentifying a dangerous object. The information of dangerous object nameis information indicating the name of a dangerous object. Theinformation of type of danger is information indicating the type ofdanger due to a dangerous object.

The information of reference position is information indicating theposition (latitude and longitude) where the dangerous object isinstalled (installation position information). Note that theinstallation position information is information indicating the positionof a fixed dangerous object and indicates the current position of thedangerous object, and hence is equivalent to dangerous object positioninformation. Further, the installation position information can also beused as information identifying a position where a dangerous range imageis displayed (display position information). Here, for example, in thecase of position information with the latitude of “35.354417” and thelongitude of “139.867623”, it is represented as position information“35.354417, 139.867623”. In this embodiment, for the convenience ofexplanation, the altitude of a dangerous object to be handled is assumedas “0”.

The information of state is information for identifying the operatingstate of a dangerous object (state identification information). In FIG.30, two types of state identification information are presented. Thefirst is state identification information (“9:00 to 17:00”, operatinghours, or the like) for identifying whether the dangerous object is inoperation or not based on the current time. The second is stateidentification information (when fully operating, when only east side isoperating, and so on) for identifying what mode the dangerous object isoperating in based on the mode of the dangerous object when it operates(or is operating).

The information of dangerous range of each state is an example ofdangerous range information and is information indicating the dangerousrange corresponding to the state of a dangerous object. The informationof current status is information indicating whether a dangerous objectis in operation or not. In this embodiment, when a dangerous object isin operation, “◯” is set by the information processing apparatus 200,and nothing is set (or it is cleared) when the dangerous object is notin operation.

FIG. 31 is a diagram illustrating an example of a table (mobile targettable 220) storing information related to dangerous objects.

The mobile target table 220 is structured to include information ofdangerous object ID, dangerous object name, type of danger, positioningreference, state, dangerous range of each state, current status, andcurrent position. Various types of information excluding the informationof current status and the information of current position are set viathe input device 295 or the like by an administrator for example. Thesetting of the information of current status and the information ofcurrent position will be described later.

The information of positioning reference is information indicating theposition where a dangerous range image is displayed (display positioninformation). The information of current position is informationindicating the current position of a dangerous object (dangerous objectposition information). Note that the descriptions of the same items asthose illustrated in FIG. 30 are omitted.

Here, in this embodiment, although the information related to dangerousobjects are stored in the form of a table and in plural tables accordingto the types of dangerous objects (whether it is a fixed dangerousobject, whether it is a moving dangerous object, or the like), it is notlimited to this structure. For example, a structure to store informationrelated to dangerous objects as one or plural files may be employed.Further, for example, a structure to store information related todangerous objects as one table may be employed.

FIG. 32 is a diagram illustrating an example of a flowchart related toprocessing of setting the “current status” (operating state settingprocessing) which is performed by an operating state setting unit, whichis an example of the function of the information processing apparatus200. Note that the processing of the flowchart, illustrated in FIG. 32is performed at certain intervals (for example, every minute).

First, the operating state setting unit obtains the current time (stepS105).

Next, the operating state setting unit reads information related to adangerous object from the storage device 280 (step S110).

Next, the operating state setting unit sets a “current status” based onthe current time and the information related to the dangerous object(step S115) and finishes the operating state setting processing.

More specifically, the operating state setting unit reads theinformation related to the dangerous object sequentially, and when theinformation of state is the state identification information (timeperiod) for identifying whether the dangerous object is in operation ornot based on the current time, the operating state setting unitdetermines whether the current time is included in this time period ornot. At this moment, the operating state setting unit sets “◯” to the“current status” when it determines that the current time is included inthis time period. On the other hand, the operating state setting unitclears the “current status” when it determines that the current time isnot included in this time period.

Here, an example of performing the processing of the flowchart of FIG.32 when the current time is “15:00” will be described.

First, in step S105, the operating state setting unit obtains thecurrent time “15:00”.

Next, in step S110, the operating state setting unit reads theinformation of state from the positionally fixed target table 210 andthe mobile target table 220.

Next, in step S115, the operating state setting unit sets the “currentstatus” corresponding to the information of state one by one. Forexample, when a time period “9:00 to 17:00” is read as the informationof state, the operating state setting unit determines whether thecurrent time “15:00” is included in the time period “9:00 to 17:00” ornot. Then, the operating state setting unit determines that the currenttime “15:00” is included in the time period “9:00 to 17:00” and sets “◯”to the “current status” corresponding to the information of state (timeperiod “9:00 to 17:00”).

Note that the processing of setting the “current status” is not limitedto the above-described processing. For example, instead of theprocessing of step S105 to step S115, the operating state setting unitmay perform processing to receive information indicating thepresence/absence of operation of the dangerous object from the operatingstate detection sensor, determine whether the dangerous object isactually operating or not, and set “◯” to the “current status” when itdetermines that the dangerous object is operating or clear the “currentstatus” when it determines that the dangerous object is not operating.

FIG. 33 is a diagram illustrating an example of a flowchart related toprocessing of setting the “current status” (operating state settingprocessing) which is performed by the operating state setting unit,which is an example of the function of the information processingapparatus 200. Note that the processing of the flowchart illustrated inFIG. 33 is performed upon reception of dangerous object stateinformation from the operating state detection sensor.

First, the operating state setting unit receives dangerous object stateinformation from the operating state detection sensor (step S120). Thedangerous object state information includes the information of dangerousobject ID and information (operating mode information) indicating themode of operation of a dangerous object. For example, as the operatingmode information, “0” is set when the dangerous object is not operating,“1” is set when the dangerous object is operating in a first mode, “2”is set when the dangerous object is operating in a second mode, and soon.

Next, the operating state setting unit reads information related to adangerous object from the storage device 280 (step S125). Morespecifically, the operating state setting unit reads information relatedto the dangerous object corresponding to the information of dangerousobject ID included in the dangerous object state information.

Next, the operating state setting unit sets the “current status” basedon the received dangerous object state information and the informationrelated to the dangerous object (step S130), and finishes the operatingstate setting processing.

More specifically, when the “state” corresponding to the operating modeinformation included in the dangerous object state information can bedetermined from the read information related to the dangerous object,the operating state setting unit sets “◯” to the “current status”corresponding to the identified “state”. On the other hand, when itcannot be determined (the case where the operating mode Informationindicates that the dangerous object is not in operation), the operatingstate setting unit clears all the read “current statuses”.

Here, an example of performing the processing of the flowchart, of FIG.33 when the dangerous object state information (dangerous object ID“001”, operating mode information “2”) is received from the operatingstate detection sensor detecting an operating state of a dangerousobject. ID “001” will be described.

First, in step S120, the operating state setting unit receives thedangerous object state information from the operating state detectionsensor detecting the operating state of a dangerous object “crane” ofthe dangerous object ID “001”.

Next, in step S125, the operating state setting unit reads states “whenfully operating” and “when only east side is operating” corresponding tothe dangerous object ID “001” from the positionally fixed target table210 and the mobile target table 220. In this embodiment, it is assumedthat as the state “when fully operating”, “1” is defined as an operationin the first mode, and as the state “when only east side is operating”,“2” is defined as an operation in the second mode.

Next, in step S130, the operating state setting unit determines thestate “when only east side is operating” corresponding to the operatingmode information “2” included in the dangerous object state informationfrom the read state “when fully operating: 1” and “when only east sideis operating: 2”, and sets “◯” to the “current status” corresponding tothe determined state “when only east side is operating”.

FIG. 34 is a diagram illustrating an example of a flowchart related toprocessing of setting the current position information of a dangerousobject. (position information setting processing) which is performed bya position information setting unit, which is an example of the functionof the information processing apparatus 200.

In the beginning, the position information setting unit receivesdangerous object determination information from a measuring instrument,measuring the position of a dangerous object (or a dangerous objecthaving a measuring instrument) (step S135). The dangerous objectdetermination information includes current position information of adangerous object (dangerous object position information) and informationof a dangerous object ID. Note that it is also possible to employ astructure such that, instead of the dangerous object positioninformation, orbit information or the like for calculating the dangerousobject position information is received, and the position informationsetting unit calculates the dangerous object position information.

Next the position information setting unit seas (updates) the “positioninformation” based on the received dangerous object position informationand information related to the dangerous object (step S140), andfinishes the position information setting processing.

For example, in step S140, when the position information setting unitreceives the dangerous object determination information including adangerous object ID “010” and dangerous object position information“35.355187, 139.874618”, the position information setting unit refers tothe positionally fixed target table 210 and the mobile target table 220,and sets the dangerous object position information “35.355187,139.874613” to the “current position” corresponding to the dangerousobject ID “010”.

FIG. 35 is a diagram illustrating an example of a flowchart related todisplay processing performed by respective functions in the AR providingapparatus 1000. It is assumed that the display processing is performedrepeatedly when a mode for identifying the presence/absence of usage ofthe AR function (AR mode) is ON. Note that the user switches ON/OFF ofthe AR mode by pressing down a switch (not illustrated) for switchingthe AR mode.

First, in step S145, the retrieval unit 1055 retrieves (extracts) targetdata of which current status is “◯” from the positionally fixed targettable 210 and the mobile target table 220 from the informationprocessing apparatus 200. Note that the target data extracted from thepositionally fixed target table 210 includes information of dangerousobject ID, reference position, and dangerous range of each state.Further, the target data extracted from the mobile target table 220includes information of dangerous object ID, positioning reference,dangerous range of each state, and current position.

In this embodiment, as illustrated in FIG. 30 and FIG. 31, four piecesof data are read as the target data of which the current status is “◯”.For example, the target data determined by the state “2 (only east sideis operating)” of the dangerous object ID “001” is read, and this targetdata include the dangerous object ID “001”, reference position“35.354417, 139.867623”, and dangerous range “radius 15 m, height 20 m,direction 0° to 180°” of each state.

Next, in step S150, the video obtaining unit 1070 obtains video data ofa real space imaged in the imaging device 1035.

Next, in step S155, the measurement unit 1040 measures information (ARposition information) indicating the current position, of the ARproviding apparatus 1000 from orbit information obtained from asatellite via the communication device 1015.

Next, in step S160, direction determination units (the directioninformation obtaining unit 1045 and the posture information obtainingunit 1050) determines the direction (direction and posture) of the ARproviding apparatus 1000. That is, the direction information obtainingunit 1045 obtains direction information calculated in the directiondetection device 1025, and the posture information obtaining unit 1050obtains posture information calculated in the posture detection device1030.

Next, in step S165, the visual field determination unit 1060 determinesthe AR visual field of the user wearing the display device 1020 based onmeasurement results (information obtained in step S150 and step S155)and so on.

Here, an example of an AR visual field U of user 400 is illustrated inFIG. 36. In this embodiment, since the altitude of the dangerous objectis set to “0”, the AR visual field is determined in a planar form (ortwo dimensionally). However, when the altitude is added to the dangerousobject position information, the AR visual field may be determined in astereoscopic form (or three dimensionally).

Further, in this embodiment, as the visual field of the user 400, forexample, information (visual field information) of the range of 160° ina leftward and rightward direction (and 120° in an upward and downwarddirection for example when the altitude is added) in set in advance andstored in the storage device 1010.

For example, the visual field determination unit 1060 determines as theAR visual field U a visual field in a predetermined range (having aradius of 100 m for example) centered on direction information 420 withreference to AR position information 410.

Next, in step S170, the display control unit 1065 generates a dangerousrange image (augmented image) of a dangerous object included in the ARvisual field based on the video data, the measurement results, and thetarget data.

More specifically, first, when the dangerous object position informationincluded in the target data is included in the AR visual field, thedisplay control unit 1065 determines that the dangerous object locatedat the dangerous object position information is a dangerous objectincluded in the AR visual field. For example, in FIG. 36, dangerousobject position information 430 is included in the AR visual field U,and hence the dangerous objects located at the dangerous object positioninformation 430 are determined as included in the AR visual field Notethat dangerous object position information 440 is not included in the ARvisual field U, and thus the dangerous object located in the dangerousobject position information 440 is determined as not included in the ARvisual field U.

Then, the display control unit 1065 obtains a geometric transformationmatrix based on the AR position information, the direction information,and the posture information.

At this moment, when the dangerous object is fixed, the display controlunit 1065 calculates coordinates (range coordinates) corresponding tothe dangerous range of each state corresponding to the determineddangerous object based on the coordinates of the dangerous objectposition information in a reference coordinate system, and applies thegeometric transformation matrix to the calculated range coordinates togenerate a dangerous range image (augmented image).

On the other hand, when the dangerous object moves, the display controlunit 1065 analyzes the video data (image data constituting the videodata) to determine the coordinates of a marker indicating a positioningreference added to the determined dangerous object in the referencecoordinate system, calculates the range coordinates corresponding to thedangerous range of each state with the determined coordinates being theorigin, and applies the geometric transformation matrix to thecalculated range coordinates so as to generate the augmented image.

Note that the method for generating an augmented image is not limited tothe above-described structure. For example, the display control unit1065 may calculate the point corresponding to the dangerous objectposition information and the point corresponding to the AR positioninformation in a virtual coordinate system based on the video data, andmay calculate a magnification from the distance between these points(relative distance) and the distance (absolute distance) between thedangerous object position information and the AR position information inthe reference coordinate system, so as to generate the augmented image.

At this moment, when the dangerous object is fixed, the display controlunit 1065 multiplies the dangerous range of each state corresponding tothe determined dangerous object by the magnification, so as to generatethe augmented image.

On the other hand, when the dangerous object moves, the display controlunit 1065 analyzes the video data (image data constituting the videodata) to determine the coordinates of the marker indicating apositioning reference added to the determined dangerous object in thereference coordinate system, applies the geometric transformation matrixto the determined coordinates to calculate coordinates (displaycoordinates) in the virtual coordinate system, and multiplies thedangerous range of each state corresponding to the determined dangerousobject by the magnification with the calculated display coordinatesbeing the origin, so as to generate the augmented image.

In step S175, the display control unit 1065 controls transmittance ofthe display device 1020 so that the augmented image is overlaid on thedangerous object in the real space (to make the dangerous objectvisually recognizable), and displays the augmented image.

In addition, when the display control unit 1065 determines pluraldangerous objects included in the AR visual field, the display controlunit generates and displays an augmented image for each of the pluraldangerous objects. Note that the display control unit 1065 may overlaythe generated augmented image on the video data and display overlaidimages on the display device 1020.

Further, the method for displaying an augmented image is not limited tothe above-described structure. For example, information (featurequantity or the like) for determining a dangerous object which isregistered in advance may be obtained based on a dangerous object IDfrom the information processing apparatus 200, and the display controlunit 1065 may perform recognition processing for video data using thefeature quantity or the like to determine a dangerous object, anddisplay a dangerous range image (augmented image) at the coordinates ofthe determined dangerous object. With this structure, the position fordisplaying the augmented image can be corrected to a position moresuitable to the real space.

FIG. 37A illustrates an example when an augmented image 510 is overlaidand displayed on a crane (dangerous object ID “001”, state “when onlyeast side is operating”) 500 as a display example of the case where adangerous object is fixed. In this example, the height is set to “20 m”in the dangerous range of each state, and thus a stereoscopic(three-dimensional) augmented image 510 is displayed, Note that when theheight is set to “0 m”, a planar (two-dimensional) augmented image 520is displayed. FIG. 37B illustrates an example when the two-dimensionalaugmented image 520 is displayed.

Further, FIG. 37C displays an example when an augmented image 610 isoverlaid and displayed on a crane truck (dangerous object ID “011”,state “moving”) 600 to which a marker 620 indicating a positioningreference is added, as a display example of the case where the dangerousobject moves.

In this embodiment, the structure to generate and display the dangerousrange image has been described, but it is not limited to this. Forexample, a mode for identifying the presence/absence of usage of asafety display function (safety display mode) may be provided, and whenthe safety display mode is ON, the display control unit 1065 may displayan image related to safety (evacuation route image or safety zone image)on the display device 1020 in addition to or instead of the dangerousrange image.

For example, the display control, unit 1065 analyzes the dangerous rangeimage to determine a part which is not the dangerous range image (arange which is not designated as the dangerous range) as a safety range,and generates an image (safety zone image) indicating the safety range.Further, for example, the display control unit 1065 calculates anevacuation route based on position information indicating the positionof an exit of the factory which is set in advance and the AR positioninformation, and generates an image (evacuation route image) indicatingthe calculated evacuation route. FIG. 37D illustrates an example when animage related to safety is displayed (evacuation route image 700).

Note that the user switches ON/OFF of the safety display mode bypressing down a switch (not illustrated) for switching the safetydisplay mode.

In step S180, the display control unit 1065 determines whether to stopthe display or not. At this moment, when the AR mode is ON the displaycontrol unit 1065 determines not to stop the display, and subsequently,the processing of step S145 is performed. On the other hand, when the ARmode is OFF, the display control unit 1065 determines to stop thedisplay, and the display processing is stopped.

FIG. 38 is a diagram illustrating an example of a flowchart related todisplay processing performed by respective functions in the AR providingapparatus 1000. This display processing is performed following theprocessing of step S175 or performed instead of the processing of stepS170 and step S175.

First, the retrieval unit 1055 retrieves traveling plan information,which is an example of movement schedule information indicating aschedule of movement of a dangerous object, from the informationprocessing apparatus 200 (step S185). The traveling plan informationincludes traveling position information indicating a position oftraveling schedule of a dangerous object and traveling time informationindicating a time thereof.

Next, in step S190, the display control unit 1065 determines a scheduleddangerous range. More specifically, the display control unit 1065determines as the scheduled dangerous range a predetermined range (forexample, the inside of a circle with a diameter of 50 m) with thedangerous object information being at its center.

Next, in step S195, the display control unit 1065 determines whether adangerous object enters the scheduled dangerous range or not based onthe obtained traveling plan information and the current time. Morespecifically, the display control unit 1065 obtains from the travelingplan information the traveling position information within a certaintime (for example, within five minutes) from the current time, anddetermines whether the obtained traveling position information isincluded in the scheduled dangerous range or not. At this moment, whenthe display control unit 1065 determines that it is included (thedangerous object enters the scheduled dangerous range), processing ofstep S200 is performed subsequently. On the other hand, when the displaycontrol unit determines that it is not included (the dangerous objectdoes not enter the scheduled dangerous range), the display processing isended. Note that the display control unit 1065 determines that it is notincluded when the dangerous object position information is included inthe AR visual field.

In step S200, the display control unit 1065 displays on the displaydevice 1020 an image indicating that the dangerous object isapproaching. More specifically, the display control unit 1065 determinesthe direction in which the dangerous object is approaching from theobtained traveling schedule position and measurement results, anddisplays on the display device 1020 an image not that the dangerousobject is approaching from the determined direction FIG. 39 illustratesan example of display when a dangerous object (AGV: Automatic GuidedVehicle) is approaching from the left side of the user (warning displayimage 800).

Note that although the display control unit 1065 determines whether thedangerous object enters the scheduled dangerous range or not, it is notlimited to this structure. For example, whether the dangerous object isapproaching the AR providing apparatus 1000 or not may be determinedbased on the current time, the traveling plan information, andmeasurement results.

Embodiment 9

In Embodiment 8, the structure in which the AR providing apparatus 1000has the imaging device 1035 is employed. An AR providing apparatus 1000according to Embodiment 9 is characterized in that it does not have theimaging device 1035. Hereinafter, a main structure which is differentfrom Embodiment 8 will be described.

FIG. 40 is a diagram illustrating a flowchart related to displayprocessing performed by respective functions in the AR providingapparatus 1000. The same reference numeral is added to the sameprocessing as that illustrated in FIG. 35 of Embodiment 8, and thedescription thereof is omitted. Note that since the AR providingapparatus 1000 does not have the imaging device 1035, the processing ofstep S150 illustrated in FIG. 35 is not performed.

In step S305, the display control unit 1065 generates a dangerous rangeimage (augmented image) of a dangerous object included in the AR visualfield based on measurement results and target data.

More specifically, first, when the dangerous object position informationincluded in target data is included in the AR visual field, the displaycontrol unit 1065 determines that the dangerous object located at thedangerous object position information is a dangerous object included inthe AR visual field.

Then the display control unit 1065 obtains the geometric transformationmatrix based on AR position information, direction information, andposture information.

Then the display control unit 1065 calculates the range coordinatescorresponding to the dangerous range of each state corresponding to thedetermined dangerous object based on the coordinates of the dangerousobject position information in the reference coordinate system, andapplies the geometric transformation matrix to the calculated rangecoordinates so as to generate the augmented image. Note that althoughthe method for generating a different augmented image depending onwhether the dangerous object is fixed or not is employed in Embodiment8, the augmented image is generated by the above-described methodregardless of whether the dangerous object is fixed or not in Embodiment9 since it is not possible to image a marker.

Other Embodiments

The above-described AR providing apparatus obtains information and so onrelated to a dangerous object so as to generate an augmented image fromthe information processing apparatus, but it is not limited to this. Forexample, the AR providing apparatus may store information fordetermining dangerous objects (feature quantities, image data ofdangerous objects, and so on) in the storage device 1010, analyze videodata imaged in the imaging device 1035 to determine a dangerous objectincluded in the video data by recognition processing and, when it isdetermined that the determined dangerous object is in repetitive motion(rotational motion, reciprocal motion, or the generate a dangerous rangeimage (augmented image) assuming the dangerous range due to thedetermined dangerous object.

Further, as the AR providing apparatus, various forms may be employed.For example, a portable terminal type AR providing apparatus may beemployed. In this case, instead of the transmissive liquid crystaldisplay, a non-transmissive liquid crystal display may be employed.Further, for example, a head-up display type AR providing apparatus maybe employed in which an augmented image is combined and displayed with atarget object in a real space which is projected opticallytransmissively.

Moreover, although functions are shared between the AR providingapparatus 1000 and the information processing apparatus 200 in theabove-described embodiments, they may be performed in one apparatus, oran apparatus different from that described in the embodiments may have asimilar function, such as generating an augmented image by theinformation processing apparatus 200 instead of the AR providingapparatus 1000.

Further, although a crane, a gas pipe, and a lathe are illustrated asthe fixed dangerous object in the above-described embodiments, it is notlimited to them. For example, a power-transmission line for transmittingelectric power may be employed as the dangerous object. At this time,the operating state detection sensor detects that it is in operation (itis dangerous) when the voltage reaches a prescribed value (prescribedvolts). Further, for example, piping for delivering fluid may beemployed as the dangerous object. At this time, the operating statedetection sensor detects that it is in operation when the temperature ofthe fluid reaches a prescribed value (prescribed temperature). Further,for example, a grinding machine (grinder) for finish-grinding thesurface of a workplace while rotating a grinding wheel may be employedas the dangerous object. At this time, the operating state detectionsensor detects that it is in operation by detecting an operating statussignal or operating schedule signal.

Further, in the above-described embodiments, when a dangerous object isrecognized by the AR providing apparatus 1000, the structure is togenerate and display an augmented image of the dangerous range of thisdangerous object, but it is not always necessary to generate and displaythe augmented image of the dangerous range for all dangerous objectsrecognized by the AR, providing apparatus 1000.

For example, when plural dangerous objects are captured in the visualfield of the AR providing apparatus 1000, and if dangerous ranges arepresented simultaneously for all the dangerous objects, it is possiblethat the user gets confused about where to pay attention. Morespecifically, it is not necessary to generate and present an augmentedimage of a dangerous range for a dangerous object which is located sofar that it is not so dangerous yet for the user at the present moment.By presenting an augmented image even for such a dangerous object whichis not so dangerous, it is rather possible that the user overlooks anaugmented image for a dangerous object which is located close and reallyneeds to be paid attention to.

Accordingly, it is possible to employ a structure to present anaugmented image illustrating a dangerous range only for a dangerousobject for which it is necessary at the present moment to display adangerous range to the user. Describing more specifically, thisstructure is as follows.

In this structure, the AR providing apparatus 1000 generates anaugmented image of the dangerous range of a dangerous object for theuser only when, for example, the position of the user is located at apredetermined distance or closer (at a predetermined threshold orcloser) from the dangerous object. FIG. 41 is a diagram illustrating anexample of a table storing information related to dangerous objects(positionally fixed target table 910). Note that in the tableillustrated in FIG. 41, additional items of display conditions are addedcompared to the table illustrated in FIG. 30.

In the structure using the positionally fixed target table 910, forexample, since the display condition of the crane of the dangerousobject ID “001” is to be within 30 m, the AR providing apparatus 1000generates an augmented image and presents it to the user only when thedistance to the crane of the dangerous object ID “001” is within 30 m.The other items are the same as those in the table illustrated in FIG.30, and Thus the descriptions thereof are omitted.

FIG. 42 is a diagram illustrating an example of a flowchart related todisplay processing. Note that in the flowchart illustrated in FIG. 42,step S405 for determining a distance between the AR providing apparatus1000 and a dangerous object is added, compared to the flowchartillustrated in FIG. 35.

In step S405, when the display control unit 1065 determines a dangerousobject included in the AR visual field, it obtains the distance to thedangerous object.

Regarding the distance between the AR providing apparatus 1000 and thedangerous object, it can be structured such that the display controlunit 1065 calculates this distance based on the information indicatingthe current position of the AR providing apparatus 1000 (AR positioninformation) and the reference position of the dangerous objectindicated in FIG. 41. Alternatively, it may be structured such that ameasurement unit is provided in the AR providing apparatus 1000, themeasurement unit emits a laser to the target dangerous object, and themeasurement unit measures the distance from the time the emitted laseris reflected at the dangerous object and returns to the measurementunit. Note that the steps other than step S405 are similar to those inthe flowchart illustrated in FIG. 35, and thus the descriptions thereofare omitted.

Further, the display control unit 1065 reads the distance stored as adisplay condition from the positionally fixed target table 910, andcompares it with the measurement result of the distance from thedangerous object. Then, when the measurement result of the distance fromthe dangerous object satisfies the display condition stored in thepositionally fixed target table 910, the processing proceeds to stepS170.

For example, in the case of the crane of the dangerous object ID “001”,when it determines a crane included in the AR visual field, the displaycontrol unit 1065 obtains the distance to the crane. Further, thedisplay control unit 1065 reads the display condition “within 30 m”corresponding to the crane of the dangerous object ID “001” from thepositionally fixed target table 910. Then, the display control unit 1065determines whether the obtained distance to the crane is “within 30 m.”or not, which is the display condition. As a result of thedetermination, when it is within 30 m, the processing proceeds to stepS170, the display control unit 1065 generates an augmented imagerepresenting a dangerous range for the dangerous object, and overlaysand displays the image. On the other hand, when it is over 30 m, theprocessing returns to step S145.

Note that although an example in which items of display conditions areadded to the positionally fixed target table illustrated in FIG. 30 isdescribed in the above-described examples, it may be structured to additems of display conditions to the mobile target table illustrated inFIG. 31 and apply the flowchart illustrated in FIG. 42 to a movingdangerous object, and hence it is not limited in particular.

Further, although the structure in the above-described embodiments is topresent information or the like related to a dangerous object to aworker, it may be a structure to provide information related to adangerous object to a guard who patrols a work site instead of theworker, and hence it is not limited in particular.

With the structures of the above-described embodiments, a dangerous spotcan be notified more appropriately when the dangerous spot changestemporally.

In the foregoing, the preferred embodiments of the present inventionhave been described detail, but the present invention is not limited tosuch certain embodiments, Various changes and modifications are possiblewithin the scope of the spirit of the present invention described in theclaims.

Note that the above-described embodiments may be combined arbitrarilyand implemented.

INDUSTRIAL APPLICABILITY

Aspects presented herein can be applied to a technology of AR displayingthe operating status of a facility or the like for an observer in aninstallation or a factory. Further, the aspects can be applied to atechnology of AR displaying a dangerous spot when the dangerous spotchanges temporally.

The invention claimed is:
 1. An information processing apparatus, comprising: an image input unit inputting an image of a facility imaged in an imaging device; a measurement information input unit inputting measurement information measured by a sensor provided in the facility from the sensor; a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input by the measurement information input unit; a display control unit overlaying and displaying the virtual image created in the creation unit and the image input by the image input unit on a display device; a control information input unit inputting control information transmitted to the facility from a control device controlling the facility; a control schedule information input unit inputting control schedule information representing contents of control scheduled to be performed on the facility; and a simulation unit performing simulation to estimate the status of the outside or inside of the facility when control represented by the control schedule information is performed based on the control schedule information input by the control schedule information input unit and the measurement information input by the measurement information input unit, wherein the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the control information input by the control information input unit and the measurement information input by the measurement information input unit and creates from numeric data representing results of the simulation a three-dimensional virtual image representing the results of the simulation, the facility is a facility in a steel plant, the sensor comprises at least one selected from a group consisting of a plurality of temperature sensors, an air pressure sensor, and a pressure sensor located inside or outside the facility, and the control information comprises at least one selected from a group consisting of blowing of oxygen or gas into the facility and charging of material into the facility.
 2. An information processing apparatus, comprising: an image input unit inputting an image of a facility imaged in an imaging device; a measurement information input unit inputting measurement information measured by a sensor provided in the facility from the sensor; a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input by the measurement information input unit; a display control unit overlaying and displaying the virtual image created in the creation unit and the image input by the image input unit on a display device; a control information input unit inputting control information transmitted to the facility from a control device controlling the facility; a determination unit determining an order number corresponding to a product in the facility, wherein the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the control information input by the control information input unit and the measurement information input by the measurement information input unit and creates a virtual image representing the order number determined by the determination unit, the facility is a facility in a steel plant, the sensor comprises at least one selected from a group consisting of a plurality of temperature sensors, an air pressure sensor, and a pressure sensor located inside or outside the facility, and the control information comprises at least one selected from a group consisting of blowing of oxygen or gas into the facility and charging of material into the facility.
 3. An information processing method executed by an information processing apparatus, the information processing method comprising: an image input step of inputting a image of a facility imaged in an imaging device; a measurement information input step of inputting measurement information measured by a sensor provided in the facility from the sensor; a creation step of creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input in the measurement information input step; a display control step of overlaying and displaying the virtual image created in the creation step and the image input in the image input step on a display device; and a control information input step of inputting control information transmitted to the facility from a control device controlling the facility; a control schedule step inputting control schedule information representing contents of control scheduled to be performed on the facility; and a simulation step performing simulation to estimate the status of the outside or inside of the facility when control represented by the control schedule information is performed based on the control schedule information input in the control schedule step and the measurement information input in the measurement information input step, wherein, in the creation step, the virtual image representing the status of the outside or inside of the facility is created based on the control information input in the control information input step and the measurement information input in the measurement information input step and is created from numeric data representing results of the simulation a three-dimensional virtual image representing the results of the simulation, the facility is a facility in a steel plant, the sensor comprises at least one selected from a group consisting of a plurality of temperature sensors, an air pressure sensor, and a pressure sensor located inside or outside the facility, and the control information comprises at least one selected from a group consisting of blowing of oxygen or gas into the facility and charging of material into the facility.
 4. An information processing method executed by an information processing apparatus, the information processing method comprising: an image inputting step of inputting an image of a facility imaged in an imaging device; a measurement information input step of inputting measurement information measured by a sensor provided in the facility from the sensor; a creation step of creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input in the measurement information input step; a display control step of overlaying and displaying the virtual image created in the creation step and the image input in the image input step on a display device; and a control information input step of inputting control information transmitted to the facility from a control device controlling the facility; a determination step determining an order number corresponding to a product in the facility, wherein, in the creation step, the virtual image representing the status of the outside or inside of the facility is created based on the control information input in the control information input step and the measurement information input in the measurement information input step, and wherein the virtual image represents the order number determined in the determination step, the facility is a facility in a steel plant, the sensor comprises at least one selected from a group consisting of a plurality of temperature sensors, an air pressure sensor, and a pressure sensor located inside or outside the facility, and the control information comprises at least one selected from a group consisting of blowing of oxygen or gas into the facility and charging of material into the facility.
 5. A computer program product comprising a non-transitory computer readable medium having control logic stored thereon for causing a computer to function as: an image input unit inputting an image of a facility imaged in an imaging device; a measurement information input unit inputting measurement information measured by a sensor provided in the facility from the sensor to a creation unit; a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input by the measurement information input unit; a display control unit overlaying and displaying the virtual image created in the creation unit and the image input by the image input unit on a display device; and a control information input unit inputting control information transmitted to the facility from a control device controlling the facility; a control schedule information input unit inputting control schedule information representing contents of control scheduled to be performed on the facility; and a simulation unit performing simulation to estimate the status of the outside or inside of the facility when control represented by the control schedule information is performed based on the control schedule information input by the control schedule information input unit and the measurement information input by the measurement information input unit, wherein the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the control information input by the control information input unit and the measurement information input by the measurement information input unit and creates from numeric data representing results of the simulation a three-dimensional virtual image representing the results of the simulation, the facility is a facility in a steel plant, the sensor comprises at least one selected from a group consisting of a plurality of temperature sensors, an air pressure sensor, and a pressure sensor located inside or outside the facility, and the control information comprises at least one selected from a group consisting of blowing of oxygen or gas into the facility and charging of material into the facility.
 6. A computer program product comprising a non-transitory computer readable medium having control logic stored thereon for causing a computer to function as: an image input unit inputting an image of a facility imaged in an imaging device; a measurement information input unit inputting measured measurement information by a sensor provided in the facility from the sensor; a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input by the measurement information input unit; a display control unit overlaying and displaying the virtual image created in the creation unit and the image input in the image input unit on a display device; and a control information input unit inputting control information transmitted to the facility from a control device controlling the facility; a determination unit determining an order number corresponding to a product in the facility, wherein the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the control information input by the control information input unit and the measurement information input by the measurement information input unit and creates a virtual image representing the order number determined by the determination unit, the facility is a facility in a steel plant, the sensor comprises at least one selected from a group consisting of a plurality of temperature sensors, an air pressure sensor, and a pressure sensor inside or outside the facility, and the control information comprises at least one selected from a group consisting of blowing of oxygen or gas into the facility and charging of material into the facility. 